Perception-based robotic manipulation system and method for automated truck unloader that unloads/unpacks product from trailers and containers

ABSTRACT

An automated truck unloader ( 10 ) for unloading/unpacking product, such as boxes or cases, from trailers and containers is disclosed. In one embodiment, a mobile base structure provides a support framework for a drive subassembly, conveyance subassembly, an industrial robot ( 56 ), a distance measurement subassembly, and a control subassembly ( 62 ). Under the operation of the control subassembly, an industrial robot ( 56 ) having a suction cup-based gripper arm selectively removes boxes from the trailer and places the boxes on a powered transportation path ( 88 ). The control subassembly ( 62 ) coordinates the selective articulated movement of the industrial robot ( 56 ) and the activation of the drive subassembly based upon a perception-based robotic manipulation system.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a National Entry (371) of International ApplicationNo. PCT/US2015/054079 filed on Oct. 5, 2015, entitled “Perception-BasedRobotic Manipulation System and Method for Automated Truck Unloader thatUnloads/Unpacks Product from Trailers and Containers”; which claims thebenefit of U.S. Patent Application No. 62/059,515 filed on Oct. 3, 2014,entitled 3D “Perception-Based Robotic Manipulation System for AutomatedTruck Unloading;” all of which are incorporated herein by reference intheir entirety.

TECHNICAL FIELD OF THE INVENTION

This invention relates, in general, to a machine for handling productsand, more particularly, to a perception-based robotic manipulationsystem and method for automated unloading and unpacking which employs anautomated truck unloader designed to unload and unpack product, such asboxes or cases, from trailers and containers.

BACKGROUND OF THE INVENTION

Loading docks and loading bays are commonly found in large commercialand industrial buildings and provide arrival and departure points forlarge shipments brought to or taken away by trucks and vans. By way ofexample, a truck may back into a loading bay such that the bumpers ofthe loading bay contact the bumpers on the trailer and a gap is createdbetween the loading bay and the truck. A dock leveler or dock platebridges the gap between the truck and a warehouse to provide a fixed andsubstantially level surface. Power moving equipment, such as forkliftsor conveyor belts, is then utilized to transport the cargo from thewarehouse to the truck. Human labor is then employed to stack the cargoin the truck. This is particularly true of the unloading of product,such as boxes or cases, from a truck, or freight container, for example.These systems are designed to maximize the amount the cargo unloadedwhile minimizing the use of human labor to both protect and extend thelife of the workforce. Reducing human labor, however, has provendifficult as the configuration and size of the boxes in the truck orfreight container cannot be easily predicted in advance. Therefore, aneed still exists for improved truck unloading systems that furtherreduce the use of human labor when unloading or unpacking product, suchas cases and boxes, from trailers and containers.

SUMMARY OF THE INVENTION

It would be advantageous to achieve a system and method for automatedunloading and unpacking of product, such as cases and boxes, that wouldenable a trailer or container to be fully unloaded using minimal or nohuman labor, thereby minimizing the time to unload the truck and theneed for human capital. It would also be desirable to enable a roboticcomputer-based solution that would address this problem, andparticularly issues with perception, by unloading and unstackingtrailers and containers with boxes and cases of varying sizes. To betteraddress one or more of these concerns, in one embodiment, an automatedtruck unloader for unloading/unpacking a plurality of product isdisclosed. A mobile base structure includes ends and a drive subassemblycoupled to the mobile base. The drive subassembly includes multiple forsteering and driving the mobile base. A conveyance subassembly isdisposed on the mobile base and includes a powered transportation pathoperable for transporting product between the ends.

An industrial robot is disposed on the mobile base to provide selectivearticulated movement of an end effector between the poweredtransportation path and a reachable space to such that the industrialrobot is operable to handle the product in the reachable space. A visualdetection subsystem, which includes a camera, is associated with theindustrial robot. In one embodiment, a distance measurement subassemblyis provided to determine presence of objects within a detection space,wherein the detection space and the reachable space at least partiallyoverlap. A control subassembly is mounted to the mobile base structurein communication with the drive subassembly, the industrial robot, thecamera, and the distance measurement subassembly, in order to coordinatethe selective articulated movement of the industrial robot and theactivation of the drive subassembly based upon the distance measurementsubassembly detecting objects within the detection space, and dimensionsof the trailer provided to the control subassembly.

The control assembly includes a memory accessible to a processor and thememory includes processor-executable instructions that, when executedcause the processor to construct a 3-D point cloud model from aplurality of first data images collected by the camera. The 3-D pointcloud model is a representation of a physical environment, whichincludes the product, of the industrial robot. The instructions alsocause the process to transform the 3-D point cloud model into a 3-Dvoxel model and then specify a search operation within the 3-D voxelmodel to identify a candidate product corner, which may belong to acandidate product of the plurality of product. The processor-executableinstructions further include an instruction for the camera to capture atleast one second data image of a local area of the physical environmentcorresponding to the candidate product corner. A dimensioning operationis specified that, based on the at least one second data image,dimensions the candidate product about the candidate product corner.Further instructions that are executed include a removal operation tounload the candidate product and calculate instructions for removing thecandidate product. In a further embodiment, a method forunloading/unpacking a product is provided as is a non-transitorycomputer readable medium having the aforementioned instructions storedthereon.

In another embodiment, an automated truck unloader forunloading/unpacking a plurality of product is disclosed. An industrialrobot, which includes a camera, is disposed configured to provideselective articulated movement of an end effector between a poweredtransportation path and a reachable space such that the industrial robotis operable to handle the product in the reachable space. A controlsubassembly is located in communication with the industrial robot andthe camera, the control subassembly coordinating the selectivearticulated movement of the industrial robot. The control assemblyincludes a memory accessible to a processor and the memory includingprocessor-executable instructions that, when executed cause theprocessor to construct a 3-D point cloud model from multiple first todata images collected by the camera. The instructions cause theprocessor to then transform the 3-D point cloud model into a 3-D voxelmodel and specify a search operation within the 3-D voxel model toidentify a candidate product corner, the candidate product cornerbelonging to a candidate product of the plurality of product. Aninstruction for the camera is provided that causes the processor tocapture at least one second data image of a local area of the physicalenvironment corresponding to the candidate product corner. The processorthen calculates, based on the at least one second data image,instructions for removing the candidate product, and updates the 3-Dvoxel model with the at least one second data image.

In a further embodiment, an automated truck unloader forunloading/unpacking a plurality of product is disclosed that includes anindustrial robot, including a camera, is configured to provide selectivearticulated movement of an end effector between a powered transportationpath and a reachable space such that the industrial robot is operable tohandle the product in the reachable space. A control assembly includes amemory accessible to a processor and the memory includingprocessor-executable instructions that, when executed cause theprocessor to construct a 3-D point cloud model from a plurality of firstdata images collected by the camera prior to transforming the 3-D pointcloud model into a 3-D voxel model. The processor-executableinstructions cause the processor to identify segmented planes within the3-D voxel model using connected component plane segmentation, identifysufficiently sized segmented planes by fitting a minimum area boundingrectangle to each set of inliers for the previously identified segmentedplanes, and identify a candidate product by selecting one of thesufficiently sized segmented planes with closed edge point to forecastedbox corner dimensions. The processor then calculates instructions forremoving the candidate product.

These systems and methodologies utilizing the present automated truckunloader therefore maximize the amount the product and cargo unloadedwhile minimizing the use of human labor to both protect and extend thelife of the workforce. These and other aspects of the invention will beapparent from and elucidated with reference to the embodiments describedhereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the features and advantages of thepresent invention, reference is now made to the detailed description ofthe invention along with the accompanying figures in which correspondingnumerals in the different figures refer to corresponding parts and inwhich:

FIG. 1 is a side elevation view with partial cross-section of oneembodiment of an automatic truck unloader utilizing a perception-basedrobotic manipulation system to unload to product within a trailer of atruck;

FIG. 2A is a side elevation view of the automatic truck unloaderillustrated in FIG. 1;

FIG. 2B is a second side elevation view of the automatic truck unloaderillustrated in FIG. 1;

FIG. 2C is a front perspective view of the automatic truck unloaderillustrated in FIG. 1;

FIG. 2D is a rear perspective view of the automatic truck unloaderillustrated in FIG. 1;

FIG. 3A is a front perspective view of one embodiment of an endeffector, which forms a portion of the automatic truck unloader;

FIG. 3B is a front elevation view of the end effector in FIG. 3A;

FIG. 3C is a side elevation view of the end effector in FIG. 3A;

FIG. 3D is a rear perspective of the end effector in FIG. 3A;

FIG. 3E is a cross-section view of the end effector along line 3E-3E ofFIG. 3C;

FIG. 4A is a side elevation view of one embodiment of an end effectorgripping a box in a first gripping position;

FIG. 4B is a side elevation view of the end effector in FIG. 4A grippinga box in a second gripping position;

FIG. 4C is a side elevation view of the end effector in FIG. 4A grippinga box in third gripping position;

FIG. 5 is a schematic block diagram of one embodiment of the automatedtruck unloader utilizing a perception-based robotic manipulation system;

FIG. 6 is a functional block diagram of one embodiment of the automatedtruck unloader in additional detail;

FIG. 7 is a functional block diagram of one embodiment of a robotcontroller which forms a portion of the automated truck unloader;

FIG. 8 is a schematic diagram of one embodiment of the perception-basedrobotic manipulation system;

FIG. 9 is flow chart depicting one embodiment of a method forperception-based robotic manipulation;

FIG. 10 is a flow chart depicting one embodiment of a method forperception-based robotic manipulation of product; and

FIG. 11 is a flow chart depicting one implementation of the methodologyin FIG. 10 in further detail.

DETAILED DESCRIPTION OF THE INVENTION

While the making and using of various embodiments of the presentinvention are discussed in detail below, it should be appreciated thatthe present invention provides many applicable inventive concepts, whichcan be embodied in a wide variety of specific contexts. The specificembodiments discussed herein are merely illustrative of specific ways tomake and use the invention, and do not delimit the scope of the presentinvention.

Referring initially to FIG. 1, therein is depicted an automated truckunloader that is schematically illustrated and generally designated 10and may be referred to as the automated truck unloader. This automatedtruck unloader 10 is utilized in systems and methods for automated truckunloading and packing of trailers, containers and the like. A tractortrailer 12 having an operator cab 14 is towing a trailer 16 having afront wall 18, two side walls 20A, 20B (best seen in FIGS. 6A through6D, for example), a floor 22, a ceiling 24, and a rear access opening 26accessible due to an open door. A bumper 28 of the trailer 16 is backedup to a loading bay 30 of loading dock 32 such that the bumper 28touches a bumper 34 of the loading bay 30. A dock plate 36 bridges thegap between the floor 22 and a deck 38 of the loading dock 32.

As will be described in further detail hereinbelow, under thesupervision of distance measurement subassembly or subassemblies thatare components of the automated truck unloader 10, the automated truckunloader 10 maneuvers and drives automatically into the trailer 16 to aposition as proximate as possible to the front wall 18. It should beappreciated that although an operator is not depicted as operating theautomated truck unloader 10, an operator may be at location 40, anoperator platform, although unnecessary. The automated truck unloader 10operates independently of an operator and an operator is only necessaryfor certain types of troubleshooting, maintenance, and the like. Atelescoping conveyor unit 42 is connected to the automated truckunloader 10. A stream of product 46, in the form standard cases or boxes46A-46H, which may be of any dimension, is being supplied by theautomated truck unloader upon removal thereof, as shown by arrow 48. Inparticular, the automated truck unloader 10 has already unloaded boxes46A through 46E, and others, for example, at the intersection of theproduct 46 proximate the front wall 18 and the floor 22. As shown, theautomated truck unloader 10 is unloading box 46 f, which will befollowed by boxes 46 g, 46 h and other boxes 46. The automated truckunloader 10 alternates between unloading the product 46 and drivingforward to create more opportunities to grip the product 46 between thefront wall 18 and the automated truck unloader 10 until the trailer 16is at least partially unloaded of product 46.

FIG. 2A through FIG. 2D depict the automated truck unloader 10 infurther detail. A mobile base 50 supports a drive subassembly 52, aconveyance subassembly 54, an industrial robot 56, a positioningsubassembly 58, a safety subsystem 60, and a control subassembly 62,which interconnects the drive subassembly 52, conveyance subassembly 54,industrial robot 56, positioning subassembly 58, and safety subsystem60. The mobile base 50 includes a front end 64 and a rear end 66 as wellas sides 68, 70, a surface 72, and an undercarriage 74.

The drive subassembly 52 is coupled to the undercarriage 74 of themobile base 50 to provide mobility. As will be discussed in furtherdetail hereinbelow, drive wheel assemblies 78, 80, are disposed on theundercarriage 74 proximate to the sides 70, 68 respectively. A universalwheel assembly 82 is disposed on the undercarriage 74 more proximate tothe rear end 66 and centered between the sides 68, 70, respectively. Incombination, wheel assemblies 78, 80, 82 provide forward and reversedrive and steering. Retractable stabilization assemblies 84, 86 are alsodisposed on the undercarriage 74 proximate to the intersection of theend 64 and side 68, the intersection of end 66 and the side 70,respectively. As alluded to, in a forward or reverse drive and steeringoperation, such as moving into or out of the trailer 16, drive wheelassemblies 78, 80 and the universal wheel assembly 82 are actuated andin contact with the deck 38 of the loading dock 32 while the retractablestabilization assemblies 84, 86 are withdrawn from contact with the deck38 in a position close to the undercarriage 74. On the other hand, whenthe automated truck unloader 10 is conducting a product loading orunloading operation, such as during the use of the industrial robot 56,the retractable stabilization assemblies 84, 86 are positioned incontact with the deck 38 to anchor the automated truck unloader 10. Itshould be appreciated that although the automated truck unloader 10 isbeing described relative to unloading and unpacking, the automated truckunloader 10 may also be used to load and pack product, including boxesand cases, into a trailer.

The conveyance subassembly 54 is disposed on the surface 72 of themobile base 50 to provide a powered transportation path 88 operable formeasuring, separating, carrying, and stacking, as required by theapplication and job assignment of the automated truck unloader 10,boxess from the rear end 66 to the front end 64 proximate to theindustrial robot 56. As shown, the powered transportation path 88includes a powered roller conveyor having roller elements 92 whichtransfer the product or boxes 46 from the industrial robot 56 to adownstream location. The conveyance subassembly 54 as well as thetelescoping conveyor unit 42 may also each be equipped with a series ofend stop photo eyes to adjust the rate of automatic flow of productthrough the telescoping conveyor unit 42 and the conveyance subassembly54. Such an implementation provides a steady and continuous flow ofproduct, maintains proper box or product separation, and preventsunnecessary gaps between the product and product backups and jams.

The industrial robot 56 is disposed at the front end 64 and adapted toprovide selective articulated movement of an end effector 130 betweenthe landing platform 94 of the powered transportation path 88 and areachable space 132 such that the industrial robot 56 is operable toplace the product 46 in the reachable space 132. The end effector 130includes a gripper arm 134 adapted for manipulating product withcooperating and complementary grapplers 136A, 136B. It should beappreciated that any type of end effector 130 may be employed theindustrial robot and the choice of end effector 130 will depend upon theproduct 46 and specific automated truck unloader 10 application. By wayof example, the gripper arm 134 with grapplers 136A, 138B is preferredfor unloading and unpacking boxes 46A-46H. It should be understood,however, that the product 46 may be any type of good such as other casedor non-cased objects requiring loading.

In one implementation, the industrial robot 56 includes seven segments130, 138, 140, 142, 144, 146, 148 joined by six joints 150, 152, 154,156, 158, 160 to furnish selective articulated movement having sixdegrees of freedom. More particularly, the referenced reachable space132, as best seen in FIGS. 2C and 2D, is defined by the movement of theindustrial robot 56 which provides rotation about six axes includingrotary movement of the entire industrial robot 56 about a primaryvertical axis; rotary movement of segment 146 having a tower structureabout horizontal axis to provide extension and retraction of the segment144 having a boom arm; rotary movement of the boom arm about thehorizontal axis to provide raising and lowering of the boom arm; andselective rotary movement about three wrist axes.

The positioning subassembly 58 is dispersed throughout the mobile base50. A distance measurement subassembly 170 disposed at the front end 64of the mobile base 50 measures distance and determines the presence ofobjects within a detection space which is located in front of the frontend 64. In one embodiment, the detection space and the reachable space132 at least partially overlap. The distance measurement subassembly 170assists the automated truck unloader 10 with forward and reversemovement and the repositioning of the automatic case loader 10 to createadditional empty reachable space 132 for the placement of the product46. Further, the distance measurement subassembly 170 assists with thecoordination and operation of the industrial robot 56. Distance andmeasurement information gathered by the distance measurement subassembly170 is provided to the control subassembly 62.

The distance measurement subassembly 170 may be a laser range findingapparatus operating on a time-of-flight measurement basis or principleor a camera system operating on an adaptive depth principle. It shouldbe appreciated, however, that other types of distance measurements arewithin the teachings of the present invention. By way of example, andnot by way of limitation, the distance measurement subassembly mayinclude a laser range finding apparatuses, cameras, ultrasonicmeasurement apparatuses, inclinometers, and combinations thereof.Similar to distance measurement subassembly 170, distance measurementsubassemblies 174, 176 are respectively disposed at the sides 68, 70.The distance measurement subassemblies 174, 176 each may include in oneembodiment, detection spaces (not illustrated) to provide measurementand distance information to the control subassembly 62 during traversemovement operations of the automated truck unloader 10.

The safety subsystem 60 is distributed and mounted to the mobile base50. The safety subsystem 60 may include a light tower which provides aquick indication of the current status of the automated truck unloader10 to an operator and a wireless operator alert system which contactspagers or cellular devices of individuals through a wireless network.Also a cage and railing may be included around the operator platform 40to provide additional safety to the operator. Emergency buttons may belocated throughout the automated truck unloader 10 to provide forinstant and immediate power down. Front safety scanners 188 and rearsafety scanners 190 may be positioned at the front end 64 and the rearend 64 to protect the automated truck unloader 10, people, and productduring a collision with an obstacle. Additionally, the front safetybumpers 188 and the rear safety bumpers 190 may include detectors thatdetect the presence of an object and cause an automatic power downduring a collision. Side safety scanners, although not illustrated, mayalso be utilized. It should be appreciated that other safety features,such as safety bumpers, may be integrated into the automated truckunloader 10.

The control subassembly 62, which is also distributed and mounted to themobile base 50, may include control station having a user interfacedisposed at the side 70 near the operator platform 76. As discussed, thedrive subassembly 52, the conveyance subassembly 54, the industrialrobot 56, the positioning subassembly 58, and the safety subassembly 60are interconnected and in communication with the control subassembly 62via a network of concealed and sheathed cables and wires. With thisarrangement, the control subassembly 62 may coordinate the manual andautomatic operation of the automated truck unloader 10. Further, avisual detection subsystem 162 is associated with the end effector suchthat the visual detection subsystem 162 captures an image of a productspace for processing by the control subassembly, as will be discussedfurther hereinbelow.

A main frame 200 is constructed of welded steel tubing includes tubularsections which provide a rectangular framework. With reference to theoperation of the drive subassembly 52 in conjunction with the mobilebase 50, the drive wheel assemblies 78, 80 and universal wheel assembly82 provide mobility along the length of the automated truck unloader 10.In addition to providing forward and reverse capability, the oneembodiment, the drive subassembly 52 may furnish a traverse drive systemproviding the capability to move the entire automated truck unloader 10perpendicular to a trailer or fixed object at the loading dock 32.

Referring now to FIGS. 4A through 4E, wherein a gripper arm 134 of theend effector 130 having grippers 136A, 136B is depicted. Moreparticularly, the gripper arm 134 includes a main frame 280 having asupport frame 281 for attachment to the industrial robot 56. A movingframe half 282 is selectively pivotally coupled to the main frame 280 bya joint 284 to provide a range of motion between approximately aparallel presentation between the main frame 280 and the moving frame282 and approximately an orthogonal presentation between the main frame280 and the moving frame 282. The joint 284 comprises hinge pairs 286,288 secured by hinge pins 290, 292. Self aligning ball bearings 294, 296are respectively located at each of the hinge pairs 286, 288 to providefor improved engagement between the main frame 280, moving frame half282 and the surface of a product, such as a case or box. A shoulderfastener 298 and ball joint end rod 300, which also couple the mainframe 280 to the moving frame half 282, actuate the selective pivotalmovement of the moving frame half 282 relative to the main frame 280.

Multiple suction cups 302 are associated with a face 303 of the mainframe 280 and, similarly, multiple suction cups 304 are associated witha face 305 of the moving frame half 282. Bulk head fittings 306 securethe multiple suction cups 304 to the moving frame half 282. Within themain frame 280, a paddle cylinder 308, straight fitting 310, and anelbow fitting 312 secure vacuum manifolds 314, 316 within the main frame280 in pneumatic communication with the suction cups 302, 304. Vacuumsensors 318, 320 are mounted in the main frame 280 proximate to thesuction cups 302. A visual detection subsystem 162 is housed in the mainframe and includes a camera assembly 322, lighting assembly 324, and acamera mount 326.

Referring now to FIG. 3A through FIG. 4C, in an unloading operation, thevacuum sensors 318, 320 senses for the presence of a box 46. In responseto detecting an object, such as a box 46 having a top T, the vacuumsensors 318, 320 actuate the vacuum manifolds 314, 316 to generate avacuum force to grip the object via the suction cups 302, 304. As shownin FIGS. 5A through 5C, the industrial robot 56 and gripper arm 134 maygrip a box 46 in any one of three positions depending on the demands ofthe unloading or unpacking operation. A bottom-front grip (FIG. 4A), atop-front grip (FIG. 4B) or a top grip (FIG. 4C) may be used.

FIG. 5 depicts one embodiment of the automated truck unloader 10 inwhich the automated truck unloader 10 is schematically depicted toinclude a computer-based architecture including a processor 350 coupledto a bus 352 having transmitter/receiver circuitry 354, outputs 356,inputs 358, memory 360, and storage 362 interconnected therewith. In oneembodiment, the control assembly 192 includes the memory 360, which isaccessible to the processor 350. The memory 360 includesprocessor-executable instructions that, when executed cause theprocessor 350 to execute instructions for unpacking or unloading boxes46 or other objects. By way of example and not by way of limitation, theinstructions may be directed to to construct a 3-D point cloud modelfrom a plurality of first data images collected by the camera. The 3-Dpoint cloud model is a representation of a physical environment, whichincludes the product, of the industrial robot. The instructions alsocause the process to transform the 3-D point cloud model into a 3-Dvoxel model and then specify a search operation within the 3-D voxelmodel to identify a candidate product corner, which may belong to acandidate product of the plurality of product. With respect to thecandidate product corner, more specifically, the instructions cause theprocessor to identify segmented planes within the 3-D voxel model usingconnected component plane segmentation before identifying sufficientlysized segmented planes by fitting a minimum area bounding rectangle toeach set of inliers for the previously identified segmented planes. Thecandidate product selected is one of the sufficiently sized segmentedplanes with closest edge point to forecasted box corner dimensions.

The processor-executable instructions further include an instruction forthe camera to capture at least one second data image of a local area ofthe physical environment corresponding to the candidate product corner.Then, the processor calculates, based on the at least one second dataimage, instructions for removing the candidate product. Morespecifically, a dimensioning operation is specified that, based on theat least one second data image, dimensions the candidate product aboutthe candidate product corner. Further instructions that are executedinclude a removal operation to unload the candidate product andcalculate instructions for removing the candidate product. Lastly, theprocessor-executable instructions update the 3-D voxel model with the atleast one second data image. These instructions may iterativelycontinue.

FIG. 6 depicts one embodiment of the automated truck unloader 10 and thecontrol signals associated therewith, which may be deployed across thecomputer architecture shown in FIG. 5, for example. The illustratedcomponents coordinate the various functions and operations of theautomated truck unloader 10. The user interface 194, operationalenvironment database 370, programmable logic controller 372, robotcontroller 374, and distance measurement subassemblies 170, 174, 176 areinterconnected. The drive subassembly 52, conveyance subassembly 54, asrepresented by control 376 for conveyors/elevators, and safetycontroller 378 are connected to the programmable logic controller 372.Finally, the industrial robot 56 is connected to the robot controller374. In one implementation, the user interface 194, operationalenvironment database 370, and programmable logic controller 372 are partof the control subassembly 62 and the robot controller 374 forms aportion of the industrial robot 56. The safety controller 358 isincluded in the safety subsystem 60 and provides operation to theaforementioned components of this subsystem.

The user interface 194 provides user control and interaction with theautomated truck unloader 10. The user interface 194 may utilize icons inconjunction with labels and/or text to provide navigation and a fullrepresentation of the information and actions available to the operator.In addition to loading operations, user interactions may be related tomaintenance, repair and other routine actions which keep the automatedtruck unloader 10 in working order or prevent trouble from arising.

The operational data environment database 370 includes data about thereachable space 132 of the industrial robot 56, stacking methodologydata, product information as well as information about the standardsizes of trailers. The product information may be stored in theoperational data environment database 350, gathered by the conveyancesubassembly 54 as previously discussed, or gained by a combinationthereof. By having the standard sizes of trailers pre-loaded, operatortime is saved from having to enter this data and performance of theautomated truck unloader 10 is improved with this additionalinformation. By way of example, Tables I & II present exemplary examplesof type of trailer data that the automatic ruck unloader 10 may utilizein determining position and product placement. It should be appreciated,however, that the automated truck unloader presented herein may operatewithin pre-loaded information such as the trailer data presented inTables I & II.

TABLE I TRAILER DIMENSIONS Inside Inside Door Inside Height HeightOpening Trailer Type Length Width Center Front Width 28′ 27′3″ 100″ 109″ 107″ 93″  (8.5 m)  (8.3 m) (2.5 m) (2.8 m) (2.7 m) (2.4 m) HighCube 45′ 44′1½″ 93″ 109″ 106″ 87″ (13.7 m) (13.4 m) (2.4 m) (2.8 m) (2.7m)   (2 m) Wedge 48′ 47′3″ 99″ 110½″ 108½″ 93″ (14.6 m) (14.4 m) (2.5 m)(2.8 m) (2.8 m) (2.4 m) Wedge

TABLE II TRAILER DIMENSIONS CONTINUED Door Opening Rear Floor CubicOverall Overall Trailer Type Height Height Capacity Width Height 28′104″ 47½″ 2029 cft 102″ 13′6″  (8.5 m) (2.6 m) (1.2 m) (57.5 cm) (2.6 m)(4.1 m) High Cube 45′ 105½″ 50″ 3083 cft  96″ 13′6″ (13.7 m) (2.7 m)(1.3 m)  (7.3 cm) (2.4 m) (4.1 m) Wedge 48′ 105″ 48½″ 3566 cft 102″13′6″ (14.6 m) (2.7 m) (1.2 m)  (101 cm) (2.6 m) (4.1 m) Wedge

The programmable logic controller 372 coordinates overall operation andswitches between various modes of operation including manual andautomatic. The programmable logic controller 372 also provides for thehigh-level calculation and coordination required during automaticoperation for items such as the current unload quadrant unloading andsteering angel calculations during automatic navigation.

The robot controller 374 controls the motions of the industrial robot 56through built in inputs and outputs wired through the industrial robot56 and the end effector 130. It should be appreciated that although aparticular architecture is presented for the control of the automaticcase loader, other architectures are within the teachings of the presentinvention. By way of example, any combination of hardware, software, andfirmware may be employed. By way of further example, the distribution ofcontrol may differ from that presented herein.

In one operation embodiment, the programmable logic controller 372accesses the dimensions of the trailer 16 from the operationalenvironment database 372. The operator 40 has indicated through the userinterface 194 which type of trailer has arrived at the docking bay 30.Alternatively, the distance measurement subassembly 170 is operable todetect this information. The distance measurement subassemblys 170, 174,176 relay distance and position data to the programmable logiccontroller 352 which uses this information to send control signals tothe robot controller 374, the drive subassembly 52, the controller 372,and the safety controller 378. Additionally, the programmable logiccontroller 372 receives control signals, which are inputs into thebehavior process, from each of these components. Constant updates andstatus information are provided to the operator 40 by the programmablelogic controller 352 through the user interface 194.

FIG. 7 depicts one embodiment of the robot controller 372 which forms aportion of the automated truck unloader 10. The essence of the robotcontrol 372 is a robot system or control program 380, which controls theindustrial robot 56. The control program 380 can be operated by theoperator 40 by means of an operating service 362 in communication withthe user interface 194 and receives input data (as well as provideinstructions, as appropriate) from the operational environmentaldatabase 370, programmable logic controller 372, and distancemeasurement subassembly 170 by means of a driver 384. It should beappreciated, that the independence of the robot controller 374 may vary.In one implementation, the robot controller 374 may be under the controlof the programmable logic controller 374. In another implementation, asillustrated, the robot controller 374 is more autonomous and may includefeatures such as direct connection to the user interface 194.

According to one embodiment, between the driver 384 and the controlprogram 380 is provided an independent data processing layer in the formof a frame program 386, which controls the robot movements, and a unit388 for automated or event-controlled strategy or behavioral selectionon the basis of the states and signals which occur. User applicationprograms, event-controlled strategy selections and sensor programs inthe frame program 386 can be programmed by the operator 40 and directedby a robot program 390, which monitors the balance and implementation ofmanual and automatic control of the industrial robot 56.

FIG. 8 depicts one embodiment of the perception-based roboticmanipulation system, wherein product is stacked in a physicalenvironment and includes various boxes labeled 46 i, 46 j, 46 k, 46 l,46 m, 46 n, 46 o, 46 p, 46 q, 46 r, and 46 s. The automated truckunloader 10 initially captures multiple data images of the product 46 asrepresented by multiple 3-D data images 400. The multiple 3-D dataimages 400 are then converted into a partial 3-D point cloud model 402and then a fully occupied 3-D voxel model 404. Product corneridentification 406 identifies the corner of virtual box 46 m-v, whichcorresponds to physical box 46 m, as the next box to be removed. Productdimensioning 408 occurs such that the industrial robot removes product46 m from the stack of product 46.

FIG. 9 depicts one embodiment of a method for perception-based roboticmanipulation. At block 420, a set number of detection poses areperformed by the industrial robot. By way of example and not by way oflimitation, the industrial robot captures twelve (12) data images fromtwelve (12) different detection poses that provide different views ofthe product. At block 422, the 3-D data images are collected from thevarious detection poses and at block 424, the 3-D data images arestitched together into a 3-D model which may be a partial model and, inone implementation, a partial 3-D point cloud data model. At block 426,areas of non-interest are filtered out within the 3-D model to create aworld space model. At block 428, the partial 3-D model is converted intoa 3-D voxel model, which in one embodiment is a fully occupied 3-D voxelmodel. At block 430, candidate product corners are identified and atblock 432 applicable filters are applied before a valid candidate isdetermined at block 434. At block 436, a local detection and collectionof a second 3-D data image is performed at the location of the validcandidate. Thereafter, the product is located at block 438 anddimensioned at block 440. At block 442, placement of the product, whichmay include removal of the product is simulated before an interferencecheck is run at block 444. At block 446, an instruction is sent to theend effector to remove the product, which, as mentioned, may be a box.At block 448, the world model formed at block 428 is updated with thelocal 3-D data image.

FIG. 10 depicts one embodiment of a method for perception-based roboticmanipulation of product. At block 460, coordinate transformations occur.In one embodiment, in order to perform box picking operations usingperception data, coordinates observed by the camera are transformed tothe reference frame used by the robotic manipulator. By way of exampleand not by way of limitation, the following coordinate system may bedefined:

-   -   The camera coordinate system [c], in which the RGB-D point cloud        data is originally expressed    -   The tool coordinate system [t], located at the end of the        manipulator kinematic chain    -   The robot coordinate system [r], located at the base of the        manipulator

A homogenous transformation matrix is defined from the camera to toolcoordinate systems, ^(c) _(t)H, and from the tool to robot coordinatesystems, rtH. The manipulator provides rtH using encoder feedback on themanipulator joints, and ^(c) _(t)H is may be obtained using a hand-eyecalibration procedure. This may be performed offline when the camera isinstalled on the system, and the resulting transformation is stored forfuture use. A point originally expressed in the camera coordinatesystem, cp, can then transformed to the robot base using Equation 1.rp= ^(c) _(t) H ^(t) _(r) HcpAt block 462, plane segmentation occurs, wherein after the points in theinput point cloud are transformed into the robot base, planes aresegmented using connected component based plane segmentation. Thisapproach takes advantage of the pixel based organizational structure ofRGB-D data to efficiently segment multiple planes in a single image. Thesegmented plane models can then be represented as:P _(i) ={a _(i) ,b _(i) ,c _(i) ,d _(i)}

where ai, bi, ci, and di are the coefficients of the ith plane whenexpressed in Hessian normal form ax+by+cz+d=0. Additionally, the jthplane inlier and kth edge point of the ith plane are defined as:P _(i,j) ^(inlier) ={x _(i,j) ,y _(i,j) ,z _(i,j)}P _(i,k) ^(edge) ={x _(i,k) ,y _(i,k) ,z _(i,k)}

Once the sets P, pinlier, and pedge, are acquired, each segmented planeis evaluated for target box face candidacy.

At block 464, plane boundary and elimination occurs wherein a fitting ofa minimum area bounding rectangle to each set of inliers for thepreviously segmented planes occurs. The purpose of this step is to aidthe target plane selection process by identifying planes that cover aninsufficient area (i.e., the plane is too small to contain the smallestpossible box face that we wish to detect). In one embodiment, the methodof rotating callipers may be used to compute the minimum area boundingrectangles in linear time, after which we remove from consideration anyplane with a bounding rectangle side length less than the smallestpossible box dimension. For purposes of this step, the lth boundingrectangle coordinate of the ith plane in clockwise order are defined as:P _(i,l) ^(rectangle) ={x _(i,l) ,y _(i,l) ,z _(i,l)}The smallest bounding rectangle side si can then be computed using:s _(i)=min{∥P _(i,1) ^(rectangle) P _(i,2) ^(rectangle) ∥,∥P _(i,2)^(rectangle) P _(i,3) ^(rectangle)∥}

In addition to eliminating planes of insufficient size, also themethodology may choose to eliminate planes with an orientation thatdeviates more than

$\frac{\pi}{4}$radians from an ideal target orientation. This target orientationdepends on the camera pose, since we expect to see mostly ‘top’ boxfaces when looking from above and ‘front’ box faces when looking forwardtoward the stacked boxes. In the case when the camera is looking downupon potential target boxes, the resulting image is expected to containthe top box face with a normal vector pointing in the general directionof the floor bounding plane normal (i.e., in an upward direction).Conversely, when the camera is looking forward toward potential targetbox, the image is expected to contain the front box face with a normalpointing in the general direction of the rear bounding plane (i.e.,pointing straight out of the container). In both cases, removed fromfurther consideration are any planes that sufficiently deviate from theideal normal. The angular deviation of plane Pi from the target planenormal Ni is computed as:

$\theta_{i} = {\arccos\frac{\left\langle {a_{i},b_{i},c_{i}} \right\rangle N_{i}}{\left. ||\left\langle {a_{i},b_{i},c_{i}} \right\rangle||||N_{i} \right.||}}$

Using Equations 6 and 7, and the minimum box side length Smin, the finaltarget plane candidacy is computed with the function:

${{candidacy}(i)} = \left\{ \begin{matrix}\begin{matrix}{true} \\\;\end{matrix} & \begin{matrix}{{{{if}\mspace{14mu}\theta_{i}} < {\pi\text{/}4}}\mspace{25mu}} \\{{{and}\mspace{14mu} s_{i}} < S_{\min}}\end{matrix} \\{false} & {{otherwise}\mspace{34mu}}\end{matrix} \right.$

At block 466, target plane selection occurs where once a set ofcandidate box face planes has been identified using the preceding steps,a final target plane and its corresponding set of inliers is chosen forfurther processing. In one embodiment of this step, the euclideandistance of the forecasted box corner location to each of the planaredge points is computed. Of the set of candidate planes not eliminatedin the previous steps, the final target plane is chosen to be that whichcontains the closest edge point to the forecasted box corner. Once thetarget face plane is selected, we are able to proceed with box vertexdetection and orientation estimation. At block 468, vertex segmentationoccurs and the final product detection is at block 470.

FIG. 11 depicts one implementation of the methodology in FIG. 10 infurther detail. At block 480, corner candidates are selected. Atdecision block 282, if the highest corner candidate in the top area isgreater than a threshold distance then the methodology advances to block484. If not, the methodology advances to block 494 as will be discussedfurther hereinbelow. In one embodiment, the threshold distance may bedefined as 24 inches (60.96 cm). Continuing to block 484, cornercandidates are eliminated that have corners more than a thresholddistance away, such as 32 inches (81.28 cm). At block 486, from theremaining corner candidates, the minimum distance from the front planethat keeps the corners within the distance and the maximum given boxsize is determined. At block 488, the highest remaining cornercandidates are kept. For example all corner candidates within the toptwo inches (5.08 centimeters) are retained.

At block 490, from the remaining candidate corners, the minimum distanceto the left wall is determined. Candidate corners are kept that arewithin a certain distance. At block 492, the bounding cuboid of theremaining corner candidate is determined. The candidate corner selectedhas the smallest Euclidian distance to the top left corner of thebounding cuboid.

Returning to block 494, if the highest corner candidate is not in thetop area threshold, then from the remaining corners the minimum distanceto the front plane is determined and corners are kept that are withinthe maximum distance of the given box or product dimensions. At block496, the highest remaining corner candidates are kept and all othercorner candidates are removed that are more than a threshold away, suchas 2 inches (5.08 cm). At block 498, from the remaining candidatecorners, the minimum distance to the left wall is determined. Candidatecorners are kept that are within a certain distance. At block 500, thebounding cuboid of the remaining corner candidate is determined. Thecandidate corner selected has the smallest Euclidian distance to the topleft corner of the bounding cuboid.

The order of execution or performance of the methods and process flowsillustrated and described herein is not essential, unless otherwisespecified. That is, elements of the methods and process flows may beperformed in any order, unless otherwise specified, and that the methodsmay include more or less elements than those disclosed herein. Forexample, it is contemplated that executing or performing a particularelement before, contemporaneously with, or after another element are allpossible sequences of execution.

While this invention has been described with reference to illustrativeembodiments, this description is not intended to be construed in alimiting sense. Various modifications and combinations of theillustrative embodiments as well as other embodiments of the invention,will be apparent to persons skilled in the art upon reference to thedescription. It is, therefore, intended that the appended claimsencompass any such modifications or embodiments.

What is claimed is:
 1. An automated truck unloader (10) forunloading/unpacking a plurality of product (46), the automated truckunloader (10) comprising: an industrial robot (56) disposed configuredto provide selective articulated movement of an end effector (134)between a powered transportation path (88) and a reachable space suchthat the industrial robot (56) is operable to handle the product (46) inthe reachable space; the powered transportation path (88) configured totransfer product (46) away from the industrial robot (56); a camera(322); a control subassembly (62) located in communication with theindustrial robot (56) and the camera (322), the control subassembly (62)coordinating the selective articulated movement of the industrial robot(56); and the control assembly (62) including a memory (360) accessibleto a processor (350), the memory (360) including processor-executableinstructions that, when executed cause the processor to: construct a 3-Dpoint cloud model (402) from a plurality of first data images (400)collected by the camera (322), the 3-D point cloud model (402) being arepresentation of a physical environment of the industrial robot (56),the physical environment including the plurality of product (46),transform the 3-D point cloud model (402) into a 3-D voxel model (404),specify a search operation within the 3-D voxel model (404) to identifya candidate product corner (406), the candidate product corner (406)belonging to a candidate product of the plurality of product (46),provide an instruction for the camera (322) to capture at least onesecond data image of a local area of the physical environmentcorresponding to the candidate product corner (406), specify adimensioning operation (408), based on the at least one second dataimage, to dimension the candidate product about the candidate productcorner (406), specify a removal operation to unload the candidateproduct, and calculate instructions for removing the candidate product.2. The automated truck unloader (10) as recited in claim 1, wherein the3-D point cloud model (402) further comprises a partial 3-D point cloudmodel.
 3. The automated truck unloader (10) as recited in claim 1,wherein the 3-D voxel model (404) further comprises a full 3-D voxelmodel.
 4. The automated truck unloader (10) as recited in claim 1,wherein the instruction to specify a dimensioning operation to dimensionthe candidate product about the candidate product corner (406) furthercomprises determining one or more features of the at least one seconddata image based on matching the at least one second data image to aknown cuboid model.
 5. The automated truck unloader (10) as recited inclaim 1, wherein the instruction to specify a dimensioning operation todimension the candidate product about the candidate product corner (406)further comprises defining the location of the candidate product basedon discontinuities in the at least one second data image.
 6. Theautomated truck unloader (10) as recited in claim 1, wherein the endeffector further comprises a suction cup-based gripper arm.
 7. Theautomated truck unloader (10) as recited in claim 1, wherein the endeffector (134) further comprises a suction cup-based gripper arm adaptedfor manipulating a box with cooperating grapplers that grip the box in agripping position selected from the group consisting of parallel to thebox and perpendicular to the box.
 8. The automated truck unloader (10)as recited in claim 1, wherein the end effector (134) further comprises:a main frame (280) having a support frame for attachment to theindustrial robot; a moving frame half (282) selectively pivotallycoupled to the main frame (280) by a joint (284) to provide a range ofmotion between approximately a parallel presentation between the mainframe (280) and the moving frame and approximately an orthogonalpresentation between the main frame and the moving frame; a firstplurality of suction cups (302, 304) associated with a face of the mainframe (280); a second plurality of suction cups (302, 304) associatedwith a face of the main frame (280); a vacuum manifold (314, 316) housedwithin the main frame in pneumatic communication with the first andsecond pluralities of suction cups (302, 304); a vacuum sensor (318,320) mounted in the main frame (280) proximate to the first plurality ofsuction cups (302, 304), the vacuum sensor (318, 320), in response todetecting an object, being configured to actuate the vacuum manifold(314, 316) and generate a vacuum force to grip the object; and thevisual detection subsystem housed in the main frame (280), the visualdetection subsystem being configured to capture an image of the productspace for processing by the control subassembly (62).
 9. An automatedtruck unloader (10) for unloading/unpacking a plurality of product, theautomated truck unloader (10) comprising: an industrial robot (56)disposed configured to provide selective articulated movement of an endeffector (134) between a powered transportation path (88) and areachable space such that the industrial robot (56) is operable tohandle the product in the reachable space; the powered transportationpath (88) configured to transfer product away from the industrial robot(56); a camera (322); a control subassembly (62) located incommunication with the industrial robot (56) and the camera (322), thecontrol subassembly (62) coordinating the selective articulated movementof the industrial robot (56); and the control assembly (62) including amemory (360) accessible to a processor (350), the memory (360) includingprocessor-executable instructions that, when executed cause theprocessor to: construct a 3-D point cloud model (402) from a pluralityof first data images (400) collected by the camera (322), the 3-D pointcloud model (402) being a representation of a physical environment ofthe industrial robot (56), the physical environment including theplurality of product (46), transform the 3-D point cloud model (402)into a 3-D voxel model (404), specify a search operation within the 3-Dvoxel model (404) to identify a candidate product corner (406), thecandidate product corner (406) belonging to a candidate product of theplurality of product (46), provide an instruction for the camera (322)to capture at least one second data image of a local area of thephysical environment corresponding to the candidate product corner(406), calculate, based on the at least one second data image,instructions for removing the candidate product, and update the 3-Dvoxel model with the at least one second data image.
 10. An automatedtruck unloader (10) for unloading/unpacking a plurality of product, theautomated truck unloader (10) comprising: an industrial robot (56)disposed configured to provide selective articulated movement of an endeffector (134) between a powered transportation path (88) and areachable space such that the industrial robot (56) is operable tohandle the product in the reachable space; the powered transportationpath (88) configured to transfer product away from the industrial robot(56); a camera (322); a control subassembly (62) located incommunication with the industrial robot (56) and the camera (322), thecontrol subassembly (62) coordinating the selective articulated movementof the industrial robot (56); and the control assembly (62) including amemory (360) accessible to a processor (350), the memory (360) includingprocessor-executable instructions that, when executed cause theprocessor (350) to: construct a 3-D point cloud model (402) from aplurality of first data images (400) collected by the camera (322), the3-D point cloud model (402) being a representation of a physicalenvironment of the industrial robot (56), the physical environmentincluding the plurality of product (46), transform the 3-D point cloudmodel (402) into a 3-D voxel model (404), identify segmented planeswithin the 3-D voxel model (404) using connected component planesegmentation, identify sufficiently sized segmented planes by fitting aminimum area bounding rectangle to each set of inliers for thepreviously identified segmented planes, identify a candidate product byselecting one of the sufficiently sized segmented planes with closededge point to forecasted box corner dimensions, and calculateinstructions for removing the candidate product.